#
# fusion.db.db
#
# Copyright 2007 Helsinki Institute for Information Technology
# and the authors.
#
# Authors: Ken Rimey <rimey@hiit.fi>
#

# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

'''
Sqlite3-based store for collections of signed key-value pairs

This is the main Python API for application developers.

The following is a quick overview of the methods of a DB instance.
First of all, creating an instance requires specifying the location of
the database.  Closing the database before you exit is a good idea.

    __init__(path=':memory:')
    close()

As a convenience, there is an interface for storing a few critical
pieces of information purely locally.  This can be used to remember
the collection ID of the user's personal workspace, miscellaneous
configuration information, and possibly also the user's private
signing keys.

    get_local_info(key) -> value or None
    set_local_info(key, value)

The main content of the database is organized as (shared) collections
of key-value pairs.  The ID of a collection is a string of 40
hexadecimal digits.

    list_collections() -> list of collection IDs

Every collection is associated with an identity, called its owner.  An
identity is a cryptographic public key.

    get_owner(collection) -> identity or None

(Querying the owner of a collection returns None if the collection was
created elsewhere and imported here but not yet synced.)

Creating a new collection requires providing an owner identity, which
must include the private signing key.  It is also possible to create a
local replica of a collection that already exists elsewhere.  This is
called importing the collection, and it only requires specifying the
collection's ID.

    create_collection(my_identity=None) -> collection
    import_collection(collection)
    remove_collection(collection)

For convenience, all methods that accept a my_identity argument allow
it to be omitted, in which case they look for an encoded identity in
the database's local info under the key "default_identity".  A new
identity is generated and stored there when the database is initially
created.

A collection simply contains key-value pairs, where keys are paths and
values are arbitrary byte strings.  A path encodes a sequence of byte
strings by joining them together using the slash character ('/') as a
separator.  Elements of the path that are deemed troublesome are
encoded in hexadecimal with a prefix of "0x".  Troublesome elements
are '', '.', '..', those that contain slashes or control characters,
and those that start with "0x".

    get(collection, key) -> value or None
    put(collection, key, value, my_identity=None)

We can think of the elements of a path other than the last as naming
folders.  A collection's application data will normally reside in just
a few top-level folders.

It is often appropriate to prevent people from overwriting each
other's data.  For this purpose, we provide the post method, which is
like put except that it appends the signing identity's fingerprint to
the key.  The effect is as if the key can have multiple values, posted
by different identities.

    post(collection, key, value, my_identity=None)

For convenience, we provide a way to access the contents of a folder
as a Python mapping, where the keys are paths relative to the folder.

    get_view(collection, base_path='', my_identity=None) -> looks like a mapping

Under the hood, operations on the view are implemented using the get,
put, and keys methods.

A collection can also act as a group, which associates named roles
with sets of identities called members.  Typical roles are "ordinary"
and "administrator".  The empty string is also a valid role,
designating guest members or visitors.

    add_member(group, role, member_identity, my_identity=None)
    remove_member(group, role, member_identity, my_identity=None)
    is_member(group, role, identity) -> boolean

    list_members(group, role) -> list of identities
    list_members(group) -> list of identity-role pairs

Unless the owner of a collection decides otherwise, he is the only one
who can write to the collection.  The add_rights method can be used to
grant rights to others on the basis of a specified group and role.

    add_rights(collection, group, role, rights, my_identity=None)
    remove_rights(collection, group, role, rights, my_identity=None)
    list_rights(collection, group, role) -> list of rights specifications

The rights specification passed to the add_rights or remove_rights
method is a path beginning with a rights keyword.  If the rights
keyword is "put" or "post", the remaining elements of the rights
specification define a key prefix, and what is granted is permission
to put or post, respectively, to keys that start with that prefix.

For example, if application data is stored under keys of the form
"data/...", permission to modify that data can be granted using a
rights specification of the form 'put/data'.  A rights specification
of the form 'put' grants unrestricted permission to modify the
collection.

A method is provided for asking if an identity has particular rights:

    has_rights(collection, rights, identity) -> boolean

Collections store administrative information in special top-level
folders (such as 'member' and 'rights'), and granting permission to
write into those folders effectively grants permission to administer
the collection.

As a special case, if a collection grants any rights at all to guests
in that same collection (which thus also acts as a group), anyone can
add themselves as a guest member.  Moreover, guest registration is
automatic upon attempting an operation that requires the guest
membership.

Finally, you can query a collection for all keys that begin with a
given key prefix.  Such keys can have any number of elements greater
than or equal to the number of elements in the key prefix.

    keys(collection, key_prefix='') -> list of keys
    query(collection, key_prefix='') -> stream of key-value pairs

The key prefix is removed from the keys appearing in the result stream.

To execute a query repeatedly and get just what is new each time, you
can fetch the database's mark value (an integer) before each query and
pass this into the subsequent query.

    get_mark() -> last_mark
    keys(collection, key_prefix='', last_mark=0) -> list of keys
    query(collection, key_prefix='', last_mark=0) -> stream of key-value pairs

The keys method orders its results in log order -- that is, according
to when the current value of the key was added to the local database.
The query method guarantees nothing about the order of its results.

A basic usage example follows.

>>> db = DB()

>>> collection = db.create_collection()
>>> group = db.create_collection()
>>> len(db.list_collections())  # The start collection is the third.
3

>>> view = db.get_view(collection, 'data')
>>> view['greeting'] = 'Hello world!'
>>> view['greeting']
'Hello world!'

>>> full_view = db.get_view(collection)
>>> full_view['data/greeting']
'Hello world!'

>>> view['color'] = 'red'
>>> view['color'] = 'blue'
>>> for k, v in view.iteritems():
...   print '%s: %s' % (k, v)
greeting: Hello world!
color: blue

>>> from fusion.core.identity import generate_identity
>>> other_guy = generate_identity()

>>> db.put(collection, 'data/shape', 'round', other_guy)
Traceback (most recent call last):
    ...
Unauthorized
>>> db.add_member(group, 'editor', other_guy)
>>> db.add_rights(collection, group, 'editor', 'put/data')
>>> db.put(collection, 'data/shape', 'square', other_guy)
>>> view.items()
[(u'greeting', 'Hello world!'), (u'color', 'blue'), (u'shape', 'square')]

>>> db.post(collection, 'comments/1234', 'Hello?', other_guy)
Traceback (most recent call last):
    ...
Unauthorized
>>> db.add_rights(collection, collection, '', 'post/comments')
>>> db.post(collection, 'comments/1234', 'This is cool!',
...         other_guy)          #doctest: +ELLIPSIS
u'comments/1234/...'
>>> for k, v in db.query(collection, 'comments'):
...   print v
This is cool!

>>> for k in db.keys(collection, 'comments'):
...   print db.get_text(collection, 'comments/' + k)
This is cool!

>>> db.close()
'''

import time

from binascii import b2a_hex as bytes_to_hex
from binascii import a2b_hex as hex_to_bytes

from fusion.db.sqlite_db import SqliteDB
from fusion.db.exceptions import *
from fusion.db.view import View

class DB(object):
    '''
    Database Instance
    '''
    def __init__(self, path=':memory:'):
        self.db = SqliteDB(path)
        self.last_mark = None   # Database mark prior to the last query.

    def close(self):
        self.db.close()

    #
    # Local info
    #

    def get_local_info(self, key):
        return self.db.get_local_info(key)

    def set_local_info(self, key, value):
        self.db.set_local_info(key, value)

    def get_default_identity(self):
        return self.db.get_default_identity()

    def set_default_identity(self, identity):
        self.db.set_default_identity(identity)

    def get_start_collection(self):
        return bytes_to_hex(self.db.get_start_collection())

    def set_start_collection(self, collection):
        self.db.set_start_collection(hex_to_bytes(collection))

    #
    # Collections
    #

    def list_collections(self):
        return map(bytes_to_hex, self.db.list_collections())

    def collection_exists(self, collection):
        return self.db.collection_exists(hex_to_bytes(collection))

    def get_owner(self, collection):
        return self.db.get_owner(hex_to_bytes(collection))

    def create_collection(self, my_identity=None):
        return bytes_to_hex(self.db.create_collection(my_identity))

    def import_collection(self, collection):
        self.db.import_collection(hex_to_bytes(collection))

    def remove_collection(self, collection, force=False):
        self.db.remove_collection(hex_to_bytes(collection), force=force)

    #
    # Data access
    #

    def get_text(self, *args, **kwargs):
        return self.get(as_text=True, *args, **kwargs)

    def get_size(self, *args, **kwargs):
        return self.get(size_only=True, *args, **kwargs)

    def get(self, collection, key,
            autoimport=False, block=False,
            timeout=None, polling_interval=0.5,
            as_text=False, size_only=False):

        assert not (as_text and size_only)

        if timeout is not None:
            deadline = time.time() + timeout

        while True:
            try:
                result = (self._get_size(collection, key) if size_only
                          else self._get(collection, key))
            except NoSuchCollection:
                if autoimport and not self.collection_exists(collection):
                    self.import_collection(collection)
                    autoimport = False
                    result = None
                else:
                    raise

            if result is not None or not block:
                return decode_text(result) if as_text else result

            if timeout is not None:
                remaining = deadline - time.time()
                if remaining <= 0:
                    return None
                polling_interval = min(remaining, polling_interval)

            time.sleep(polling_interval)

    def _get(self, collection, key):
        c = hex_to_bytes(collection)
        key = path_to_key(key)
        return self.db.get(c, key)

    def _get_size(self, collection, key):
        c = hex_to_bytes(collection)
        key = path_to_key(key)
        return self.db.get_size(c, key)

    def put(self, collection, key, value, my_identity=None):
        c = hex_to_bytes(collection)
        key = path_to_key(key)
        value = encode_text(value)
        self.db.put(c, key, value, my_identity)

    def post(self, collection, key, value, my_identity=None):
        c = hex_to_bytes(collection)
        key = path_to_key(key)
        value = encode_text(value)
        key = self.db.post(c, key, value, my_identity)
        return key_to_path(key)

    def get_text_view(self, *args, **kwargs):
        return self.get_view(as_text=True, *args, **kwargs)

    def get_view(self, collection, base_path='', my_identity=None,
                 as_text=False):
        my_identity = my_identity or self.get_default_identity()
        return View(self, collection, base_path, my_identity,
                    as_text=as_text)

    #
    # Groups
    #

    def add_member(self, group, role, member_identity, my_identity=None):
        g = hex_to_bytes(group)
        self.db.add_member(g, role, member_identity, my_identity)

    def remove_member(self, group, role, member_identity, my_identity=None):
        g = hex_to_bytes(group)
        self.db.remove_member(g, role, member_identity, my_identity)

    def is_member(self, group, role, identity):
        g = hex_to_bytes(group)
        return self.db.is_member(g, role, identity)

    def list_members(self, group, role=None):
        g = hex_to_bytes(group)
        return self.db.list_members(g, role)

    #
    # Rights
    #

    def add_rights(self, collection, group, role, rights, my_identity=None):
        c = hex_to_bytes(collection)
        g = hex_to_bytes(group)
        rights = path_to_key(rights)
        self.db.add_rights(c, g, role, rights, my_identity)

    def remove_rights(self, collection, group, role, rights, my_identity=None,
                      retroactive=False):
        c = hex_to_bytes(collection)
        g = hex_to_bytes(group)
        rights = path_to_key(rights)
        self.db.remove_rights(c, g, role, rights, my_identity, retroactive)

    def list_rights(self, collection, group, role):
        c = hex_to_bytes(collection)
        g = hex_to_bytes(group)
        return map(key_to_path, self.db.list_rights(c, g, role))

    def has_rights(self, collection, rights, identity=None):
        c = hex_to_bytes(collection)
        rights = path_to_key(rights)
        return self.db.has_rights(c, rights, identity)

    #
    # Queries
    #

    def get_mark(self):
        return self.db.get_mark()

    def get_last_mark(self):
        return self.last_mark

    def keys(self, collection, key_prefix='', last_mark=0, reverse=False,
             autoimport=False, block=False,
             timeout=None, polling_interval=0.2):

        if timeout is not None:
            deadline = time.time() + timeout

        while True:
            try:
                keys = self._keys(collection, key_prefix, last_mark, reverse)
                if keys:
                    return keys
            except NoSuchCollection:
                if autoimport and not self.collection_exists(collection):
                    self.import_collection(collection)
                    autoimport = False
                else:
                    raise

            if not block:
                return []

            if timeout is not None:
                remaining = deadline - time.time()
                if remaining <= 0:
                    return []
                polling_interval = min(remaining, polling_interval)

            time.sleep(polling_interval)

    def _keys(self, collection, key_prefix='', last_mark=0, reverse=False):
        c = hex_to_bytes(collection)
        key_prefix = path_to_key(key_prefix)

        def fetch():
            self.last_mark = self.get_mark()
            return self.db.keys(c, key_prefix, last_mark=last_mark,
                                reverse=reverse)

        return map(key_to_path, self.db.transact(fetch))

    def query_text(self, *args, **kwargs):
        return self.query(as_text=True, *args, **kwargs)

    def query(self, collection, key_prefix='', last_mark=0,
              autoimport=False, block=False,
              timeout=None, polling_interval=0.2,
              as_text=False):

        if timeout is not None:
            deadline = time.time() + timeout

        while True:
            hits = 0
            try:
                q = self._query(collection, key_prefix, last_mark, as_text)
                try:
                    for result in q:
                        hits += 1
                        yield result
                finally:
                    q.close()
            except NoSuchCollection:
                if autoimport and not self.collection_exists(collection):
                    self.import_collection(collection)
                    autoimport = False
                else:
                    raise

            if hits or not block:
                break

            if timeout is not None:
                remaining = deadline - time.time()
                if remaining <= 0:
                    break
                polling_interval = min(remaining, polling_interval)

            time.sleep(polling_interval)

    def _query(self, collection, key_prefix='', last_mark=0, as_text=False):
        c = hex_to_bytes(collection)
        key_prefix = path_to_key(key_prefix)
        q = self.db.query(c, key_prefix, last_mark=last_mark)
        try:
            self.last_mark = self.get_mark()
            hits = 0
            for k, v in q:
                hits += 1
                if hits == 1:
                    self.last_mark = self.get_mark()

                if as_text:
                    v = decode_text(v)

                yield key_to_path(k), v
        finally:
            q.close()

    def query_all_collections_and_keys(self, last_mark=0):
        q = self.db.query_all_collections_and_keys(last_mark)
        try:
            self.last_mark = self.get_mark()
            hits = 0
            for c, k in q:
                hits += 1
                if hits == 1:
                    self.last_mark = self.get_mark()

                yield bytes_to_hex(c), key_to_path(k)
        finally:
            q.close()

def key_to_path(key):
    return '/'.join(map(hex_encode_for_path, key))

def path_to_key(path):
    path = path.strip('/')
    if not path:
        return []
    else:
        return map(hex_decode, path.split('/'))

def hex_encode_for_path(s):
    if '/' in s or s in ('', '.', '..'):
        return hex_encode(s)
    else:
        return hex_encode_for_text(s)

control_chars = set(range(1, 0x20) + range(0x7f, 0xa0))
allowed_chars = set((ord('\t'), ord('\r'), ord('\n')))
map_bad_chars_to_nulls = dict((c, 0) for c in control_chars - allowed_chars)

def hex_encode_for_text(s):
    # Hex-encode s if it starts with "0x", is not valid UTF-8, or
    # includes unicode control characters other than tab, return,
    # and newline.
    if s.startswith('0x'):
        return hex_encode(s)

    try:
        t = s.decode('utf-8')
    except UnicodeError:
        return hex_encode(s)

    if u'\x00' in t.translate(map_bad_chars_to_nulls):
        return hex_encode(s)
    else:
        return t

def hex_encode(s):
    return u'0x' + bytes_to_hex(s)

def hex_decode(t):
    if t.startswith('0x'):
        return hex_to_bytes(t[2:])
    else:
        return encode_text(t)

def encode_text(t):
    if isinstance(t, unicode):
        return t.encode('utf-8')
    else:
        return t

def decode_text(s):
    if s is None:
        return None

    try:
        return s.decode('utf-8')
    except UnicodeError:
        return u''

if __name__ == '__main__':
    import doctest
    doctest.testmod()
